At Fanout, we’ve developed a powerful publish-subscribe system for fronting custom APIs. Sometimes, though, API design is too much to think about and all you want to do is push some JSON.

To address this need, we initially created our own proprietary JSON-publishing protocol (a.k.a. “FPP”) and corresponding client library. This made it possible for developers to implement realtime updates in just a few lines of code, transported by magic. While this system worked well enough, it wasn’t terribly satisfying to have invented yet-another-protocol in order to accomplish this. Sure, every other pubsub cloud service has done the same thing, but the status quo makes for a bunch of redundant efforts that all achieve more-or-less the same result, and it hampers interoperability. So, lately, we’ve been looking into how we could adapt an existing standard for pushing JSON.

Bayeux was one option. The protocol is dated, but it gets the job done, and a modern implementation such as Faye has a JavaScript client API that is almost exactly the same as ours. Other pubsub options included WAMP and Meteor’s DDP. Finally, one could run pubsub “over” a streaming layer, such as is the case with XMPP PubSub or Socket.io rooms. For now, we chose XMPP PubSub. This isn’t to say we couldn’t support additional delivery mechanisms in the future though. UPDATE: We support Bayeux now also.

So, why XMPP PubSub? Well, it is arguably the “most standard” of the available options, in that it is well-specified and has a large number of implementations written by different authors. It also provides a clean method of access for non-browser clients. Of course, access by browsers is possible as well. Further, we were already operating XMPP facilities within the Fanout cluster (using Tigase), so most of the groundwork was in place. It may seem counterintuitive to use an XML-based protocol like XMPP to deliver JSON, but that’s just a transport detail and shouldn’t be a deterrent to an otherwise great protocol.

How we did it

The way it works is Fanout Cloud has an XMPP PubSub service at the address pubsub.fanout.io, and every Data Messaging channel is given a node of the form /{realm_id}/{channel}/. When JSON data is published via the Fanout Cloud API, the data is delivered to any subscribers by enveloping the JSON inside of an event notification. That’s really it, nothing terribly fancy:

<message from='pubsub.fanout.io' to='alice@example.com' type='headline'>
  <event xmlns='http://jabber.org/protocol/pubsub#event'>
    <items node='/d04fd503/test'>
      <item id='3ebc205b-ce77-46c3-843a-49e203e7945b'>
        <json xmlns='urn:xmpp:json:0'>
          {"foo": "bar"}
        </json>
      </item>
    </items>
  </event>
</message>

Everything here is standard XMPP and XEP-60, and the meaning of the JSON is up to your application.

Standards for the win

By using standard protocols this way, it becomes possible to use existing libraries to access the service. Here’s an example of how to use the SleekXMPP Python library to receive updates from Fanout Cloud:

import sys
import json
import logging
import sleekxmpp

realm = sys.argv[1]
channel = sys.argv[2]

class MyClient(sleekxmpp.ClientXMPP):
    def __init__(self, jid, password):
        super(MyClient, self).__init__(jid, password)
        self.register_plugin('xep_0060')
        self.add_event_handler('session_start', self.session_start,
            threaded=True)
        self.add_event_handler('pubsub_publish', self.pubsub_publish)

    def session_start(self, event):
        logging.info('started')
        self.send_presence(pto='pubsub.fanout.io')
        node = '/%s/%s' % (realm, channel)
        try:
            self['xep_0060'].subscribe('pubsub.fanout.io', node)
            logging.info('subscribed to %s' % node)
        except:
            logging.error('failed to subscribe')
            self.disconnect()

    def pubsub_publish(self, msg):
        xjson = msg['pubsub_event']['items']['item']['payload']
        data = json.loads(xjson.text)
        logging.info('event: %s' % data)

logging.basicConfig(level=logging.INFO, format='%(levelname)-8s %(message)s')

xmpp = MyClient('anon.fanout.io', None)
if xmpp.connect():
    xmpp.process(block=True)
else:
    print 'Unable to connect'

Here we’re using Fanout Cloud’s anonymous login service at anon.fanout.io. We send presence to the PubSub service as this is required for temporary subscriptions to work (long-term subscriptions are not allowed yet). The subscription is cleared when the client logs off.

Notice how, other than the service names, the code is devoid of any Fanout-isms. Also notice how there’s no XML wrangling either. SleekXMPP hides this pretty well. Easy, standards-based JSON push!

From the browser

Fanout runs a BOSH gateway so that the same service is accessible from the browser via a client such as Strophe. Here’s an example:

var XMPP_DOMAIN = 'anon.fanout.io';
var BOSH_SERVICE = 'http://' + XMPP_DOMAIN + '/http-bind/';
var PUBSUB_JID = 'pubsub.fanout.io';
var NS_PUBSUB = 'http://jabber.org/protocol/pubsub';
var connection = null;

function log(msg)
{
    $('#log').append('<div></div>').append(document.createTextNode(msg));
}

function onConnect(status)
{
    if (status == Strophe.Status.CONNECTING) {
        log('Strophe is connecting.');
    } else if (status == Strophe.Status.CONNFAIL) {
        log('Strophe failed to connect.');
        $('#connect').get(0).value = 'connect';
    } else if (status == Strophe.Status.DISCONNECTING) {
        log('Strophe is disconnecting.');
    } else if (status == Strophe.Status.DISCONNECTED) {
        log('Strophe is disconnected.');
        $('#connect').get(0).value = 'connect';
    } else if (status == Strophe.Status.CONNECTED) {
        log('Strophe is connected. JID=' + connection.jid);

        var realm = $('#realm').get(0).value;
        var channel = $('#channel').get(0).value;
        var pubsubNode = '/' + realm + '/' + channel;

        log('Subscribing to ' + PUBSUB_JID + ', node=' + pubsubNode);
        connection.send($pres({to: PUBSUB_JID}).tree());

        connection.addHandler(onMessage, null, 'message', null, null,  null);

        var reqId = connection.getUniqueId();
        var req = $iq({to: PUBSUB_JID, type: 'set', id: reqId})
            .c('pubsub', {xmlns: NS_PUBSUB})
            .c('subscribe', {node: pubsubNode, jid: connection.jid});
        connection.sendIQ(req, onSubscribeResult, onSubscribeError, 30000);
    }
}

function onSubscribeResult(resp) {
    log('Subscribed.');
}

function onSubscribeError(resp) {
    log('Failed to subscribe.');
}

function onMessage(msg) {
    var xjson = msg.getElementsByTagName('json');
    if (xjson.length > 0) {
        var data = JSON.parse(Strophe.getText(xjson[0]));
        log('Event: ' + data);
    }

    // we must return true to keep the handler alive.  
    // returning false would remove it after it finishes.
    return true;
}

$(document).ready(function () {
    connection = new Strophe.Connection(BOSH_SERVICE);

    $('#connect').bind('click', function () {
        var button = $('#connect').get(0);
        if (button.value == 'connect') {
            button.value = 'disconnect';
            connection.connect(XMPP_DOMAIN, '', onConnect);
        } else {
            button.value = 'connect';
            connection.disconnect();
        }
    });
});

Assuming the above code exists in a file called strophe_test.js, it should be possible to use it with this HTML:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"">
  <head>
    <title>Strophe.js Test</title>
    <script type='text/javascript'
      src='http://ajax.googleapis.com/ajax/libs/jquery/1.2.6/jquery.min.js'>
    </script>
    <script type='text/javascript'
      src='strophe.js'></script>
    <script type='text/javascript'
      src='strophe_test.js'></script>
  </head>
  <body>
    <div id='login' style='text-align: center'>
      <form name='cred'>
        <label for='realm'>Realm ID:</label>
        <input type='text' id='realm' />
        <label for='channel'>Channel:</label>
        <input type='text' id='channel' value='test' />
        <input type='button' id='connect' value='connect' />
      </form>
    </div>
    <hr />
    <div id='log'></div>
  </body>
</html>

The above example is a bit verbose compared to just using our own proprietary JavaScript client, but if you enjoy standards, there you go. And if we enable more XMPP services within Fanout Cloud in the future, then this becomes a much more powerful entrypoint.

Federation?

Another cool aspect of offering access via XMPP is that any address on the internet can subscribe to channels, even those client addresses not hosted by Fanout Cloud. For example, you could use a GMail account, or even your own XMPP server, to receive published data.

dmxmpp

We’re not quite sure what scenarios this may be useful in, but as XMPP is naturally a decentralized, federated system, this “just works”. Most likely, though, especially if you’re shipping an application to be used by someone else, you’ll want to use an account hosted by our service.

Conclusion

Our adoption of XMPP PubSub means that Fanout Cloud’s JSON push becomes accessible on pretty much every platform and operating system imaginable, without us having to write any new client libraries. And if you’re already an XMPP aficionado, Fanout Cloud just got that much easier to use.

Enjoy!